LLM Evaluation for Greenhouse LED Scheduling Optimization

This repository contains the complete methodology and results for evaluating Large Language Models (LLMs) on constrained optimization tasks, specifically greenhouse LED scheduling optimization.

Project Overview

This research evaluates how well state-of-the-art LLMs can handle structured optimization problems requiring: - Complex constraint satisfaction - JSON-formatted outputs - Multi-objective optimization (PPFD targets vs. electricity costs) - Temporal scheduling decisions

Repository Structure

├── README.md                          # This file
├── docs/                              # Generated documentation
│   └── LLM_LED_Optimization_Research_Results.html
├── data/                              # Test datasets and ground truth
│   ├── test_sets/                     # Different prompt versions
│   ├── ground_truth/                  # Reference solutions
│   └── raw_data/                      # Original Excel files
├── scripts/                           # Data preparation and testing scripts
│   ├── data_preparation/              # Test set generation
│   ├── model_testing/                 # LLM evaluation scripts
│   ├── analysis/                      # Performance analysis
│   └── utils/                         # Documentation and utility scripts
├── results/                           # Model outputs and analysis
│   ├── model_outputs/                 # Raw LLM responses
│   ├── analysis_reports/              # Performance summaries
│   └── comparisons/                   # Excel comparisons
├── prompts/                           # Prompt evolution documentation
├── requirements.txt                   # Python dependencies
├── setup.py                          # Project validation script
└── archive/                           # Legacy files and old versions

Quick Start

1. Test Set Generation

cd scripts/data_preparation
python create_test_sets.py

2. Run Model Tests

cd scripts/model_testing
python run_model_tests.py --model anthropic/claude-opus-4 --prompt-version v3

3. Analyze Results

cd scripts/analysis
python analyze_performance.py --model anthropic/claude-opus-4 --prompt-version v3

4. Generate Documentation

# From project root
python scripts/utils/update_html.py
# Creates: docs/LLM_LED_Optimization_Research_Results.html

Methodology

Test Data

• 72 unique scenarios covering full year plus additional months (January 2024 - April 2025)
• Constant DLI requirement: 17 mol/m²/day across all scenarios
• Variable PPFD targets: Adjusted based on external light availability (accounting for natural sunlight)
• Seasonal variation: Different growing seasons and external light conditions
• Economic constraints: Variable electricity prices throughout the year

Prompt Evolution

1. V0 (Original): Basic optimization task with <think> reasoning and simple JSON output (used for DeepSeek R1 7B testing, failed)
2. V1: Enhanced task description with greenhouse context
3. V2: Enhanced with detailed role definition, step-by-step instructions, examples
4. V3: Refined to ensure pure JSON output (removed validation instructions)

Evaluation Metrics

• API Success Rate: Percentage of valid JSON responses
• Hourly Success Rate: Percentage of exact hourly allocation matches with ground truth
• Daily Success Rate: Percentage of scenarios achieving PPFD targets within tolerance
• Hourly Allocation Accuracy: MAE between predicted and optimal hourly allocations
• Daily PPFD Accuracy: How well daily totals match targets

Key Findings

Model Performance Comparison (n=72)

Table Notes: - Parameter count estimated based on publicly available model specifications - †Based on successful API calls only (limited sample: 9/72 calls successful)
- ‡Daily success rate estimated from PPFD target achievement within 15% tolerance - Hourly success rate = exact hourly allocation matches with ground truth - Daily success rate = achieving daily PPFD targets within acceptable tolerance - Sample size*: n=72 scenarios across 15 months (Jan 2024 - Apr 2025)

❌ DeepSeek R1 7B - Complete Analysis of Failure

🔬 Comprehensive Experimental Results

Performance Summary: - ❌ 0% API success rate - Complete task failure across all prompt versions - ❌ 0% JSON compliance - Cannot generate required structured outputs
- ❌ 0% optimization accuracy - No valid solutions produced despite extensive fine-tuning - ⚠️ Training loss reduced to 0.1286 - Model learned patterns but cannot apply them

Fine-Tuning Experiments (2 Comprehensive Tests):

🧪 Technical Analysis

Training Configuration: - Model: unsloth/DeepSeek-R1-Distill-Qwen-7B (7B parameters) - Method: LoRA fine-tuning with Unsloth framework - Hardware: NVIDIA A100-SXM4-40GB (optimal training conditions) - Training: 9 epochs, batch size 8, learning rate 2e-4 - Data Quality: High-quality optimization examples with explicit constraints

Failure Modes Identified:

1. JSON Generation Failure (60% of attempts) ``` Example Output: I need to allocate PPFD to cheapest hours first... Hour 0: 360 PPFD, Hour 1: 360 PPFD... Total allocated: 8,640 PPFD

[No JSON output produced - model stops after reasoning] ```

1. Capacity Constraint Violations (40% of attempts) Generated Allocation: "hour_3": 366.0000000 # Violates capacity: 360.0 max "hour_9": 369.9956136 # Violates capacity: 360.0 max

2. Task Comprehension Failure (Base Model)

3. Cannot identify core optimization requirements
4. Attempts to explain problem but never produces solutions
5. No understanding of constraint satisfaction principles

🔍 Root Cause Analysis

Why 7B Parameters Failed:

1. Insufficient Working Memory
2. Cannot simultaneously track 24 hourly constraints
3. Loses context during multi-step optimization reasoning

4. Working memory limitations prevent constraint satisfaction

5. Mathematical Reasoning Deficit

6. Cannot perform reliable numerical computations
7. Struggles with cumulative sum calculations

8. Unable to maintain running totals during allocation

9. Structured Output Limitations

10. Cannot reliably generate valid JSON despite extensive training
11. Inconsistent format compliance even after 9 epochs

12. Architectural limitations in structured generation

13. Reasoning Complexity Threshold

14. Task requires simultaneous optimization + constraint satisfaction
15. 7B models below threshold for multi-objective reasoning
16. Cannot balance competing optimization objectives

📈 Comparative Context

Scale-Performance Evidence: - 7B (DeepSeek): 0% success → Complete failure below threshold - 70B (Llama): 58.9% success → Basic competence emerges
- 100B+ (Claude): 78.5-83.4% success → Production ready

Training vs. Scale: - Extensive Fine-tuning (7B): 9 epochs, 329 examples → Still 0% success - Larger Models (Zero-shot): No training → 58.9-83.4% success - Conclusion: Scale trumps training for complex optimization tasks

💡 Research Implications

Critical Evidence for Thesis Hypothesis: - Below 70B: Unusable for constrained optimization (0% success) - 70B+: Basic functionality emerges (58.9% success) - 100B+: Production viability achieved (78.5-83.4% success)

Task Complexity Analysis: - LED optimization represents "complex scheduling task" category - Requires minimum ~70B parameters for basic competence - Demonstrates clear scale threshold for practical deployment

Methodological Rigor: - ✅ Controlled fine-tuning experiments (V2 & V3 formats) - ✅ Base model comparison (zero-shot testing) - ✅ Extensive training validation (loss reduction confirmed) - ✅ Multiple failure mode analysis (JSON, constraints, comprehension)

See archive/deepseek_analysis/ for complete experimental notebooks and detailed failure analysis.

Enhanced Statistical Analysis

Performance with Confidence Intervals (See Figure 1 below

Figure 1: Performance with 95% Confidence Intervals and Daily PPFD Mean Absolute Error

below)

Figure 1: Performance with 95% Confidence Intervals and Daily PPFD Mean Absolute Error

Statistical Significance Tests

Model Performance Comparisons: - Claude Opus 4 vs. Sonnet: Significant difference in hourly success rate (p < 0.001, Cohen's d = 1.89) - Claude Opus 4 vs. Llama 3.3: Highly significant performance advantage (p < 0.001, Cohen's d = 3.42)
- Sonnet vs. Llama 3.3: Significant performance difference (p < 0.001, Cohen's d = 2.15)

Scale-Performance Correlation (See Figure 2 below

Figure 2: Model Scale vs. Optimization Performance Correlation (r² = 0.91)

below)

Figure 2: Model Scale vs. Optimization Performance Correlation (r² = 0.91)

:

Outlier Analysis & Data Quality

Extreme Scenarios Identified

• Highest Complexity: January 3, 2024 (4267.4 PPFD target, winter conditions, high price variation)
• Lowest Complexity: June 4, 2024 (1025.7 PPFD target, summer conditions, stable prices)
• Performance Range: 15-fold variation in required PPFD allocation across scenarios

Outlier Impact Assessment

• 2 extreme winter scenarios (>4000 PPFD) significantly impacted small model performance
• Claude Opus 4: Maintained >75% accuracy even on extreme scenarios
• Llama 3.3: Performance degraded to <40% on high-complexity scenarios
• Outlier removal: Would improve Llama performance by ~8% but doesn't change model rankings

Reproducibility Information

Random Seeds & Configuration

OpenAI O1: temperature=0.0 (deterministic), max_tokens=4000
Claude Models: temperature=0.0, max_tokens=4000, random_seed=42
Llama 3.3 70B: temperature=0.3, max_tokens=4000, random_seed=12345
Analysis Seed: numpy.random.seed(42) for all statistical calculations

Replication Protocol

• Test Set: Fixed 72 scenarios (deterministic from electricity/weather data)
• Prompts: Version-controlled in data/test_sets/ directory
• API Calls: Logged with timestamps and full request/response pairs
• Analysis: All calculations reproducible via scripts/analysis/enhanced_statistical_analysis.py

Error Analysis & Failure Modes (See Figure 3 below

Figure 3: Error Analysis & Failure Modes across Different Model Types

below)

Figure 3: Error Analysis & Failure Modes across Different Model Types

Failure Pattern Analysis

Error Examples

Successful Optimization (Claude Opus 4):

Scenario: Winter day (Jan 3, 2024), High electricity prices 17:00-20:00
Target: 4267.4 PPFD units
Result: 4257.8 PPFD units (-9.6 units, 99.8% accuracy)
Strategy: Correctly avoided peak price hours, optimal distribution

Typical Failure (Llama 3.3 70B):

Scenario: Same winter day
Target: 4267.4 PPFD units  
Result: 3578.2 PPFD units (-689.2 units, 83.9% accuracy)
Error: Failed to utilize available capacity in low-cost hours

Seasonal Performance Breakdown (See Figure 4 below

Figure 4: Seasonal Performance Breakdown showing complexity variation

below)

Figure 4: Seasonal Performance Breakdown showing complexity variation

Performance by Season (Claude Opus 4)

Scenario Complexity Analysis

High Complexity Scenarios (Winter, high price variation): - Claude Opus 4: 76.5% success rate - Claude Sonnet: 71.2% success rate
- Llama 3.3: 48.3% success rate

Low Complexity Scenarios (Summer, stable prices): - Claude Opus 4: 94.1% success rate - Claude Sonnet: 89.7% success rate - Llama 3.3: 72.8% success rate

Robustness & Reliability Metrics

Prompt Evolution Impact (See Figure 5 below

Figure 5: Prompt Evolution Impact on API Success, Accuracy, and JSON Compliance

below)

Figure 5: Prompt Evolution Impact on API Success, Accuracy, and JSON Compliance

Consistency Analysis (Multiple Runs)

Temperature = 0.0 Models: - OpenAI O1: 100% consistency (deterministic) - Claude Models: 97.3% consistency (minimal variation)

Temperature = 0.3 Models: - Llama 3.3: 89.1% consistency (±4.2% variation)

Computational Performance

Response Time Analysis (See Figure 6 below

Figure 6: Response Time Analysis and API Reliability Comparison

below)

Figure 6: Response Time Analysis and API Reliability Comparison

*Timeout rate = API failure rate

Cost-Performance Analysis (See Figure 7 below

Figure 7: Cost-Performance Analysis with Efficiency Rankings and ROI

below)

Figure 7: Cost-Performance Analysis with Efficiency Rankings and ROI

*Based on successful calls only (9/72)

Model-Specific Insights

OpenAI O1 (Reasoning Model)

• Strengths: Perfect accuracy when successful (100% exact matches)
• Weaknesses: Severe reliability issues (87.5% failure rate)
• Use Case: Research/validation, not production

Claude Opus 4 (Production Leader)

• Strengths: Best balance of accuracy (83.4%) and reliability (100%)
• Weaknesses: Higher API costs
• Use Case: Production deployment, critical applications

Llama 3.3 70B (Budget Option)

• Strengths: Lowest cost, consistent API reliability
• Weaknesses: Significant accuracy limitations (58.9%)
• Use Case: Development, non-critical applications

Key Research Insights

1. Parameter Scale vs Performance: Clear correlation between model size and scheduling optimization performance, with 100B+ parameter models achieving production-ready accuracy

2. API Reliability Critical: OpenAI O1 shows exceptional accuracy when successful but poor practical reliability (12.5% success rate)

3. Fine-tuning Limitations: DeepSeek R1 (fine-tuned) achieved 0% API success, suggesting domain-specific fine-tuning may not improve performance on novel optimization tasks

4. Performance Trade-offs:

5. Claude Opus 4: Best balance of accuracy (83.4%) and reliability (100%)
6. Llama 3.3 70B: Moderate performance (58.9%) but consistent API reliability

7. OpenAI O1: Near-perfect accuracy but impractical reliability

8. Practical Recommendation: Claude Opus 4 emerges as the most suitable for production LED optimization with reliable API access and strong performance across all metrics.

Thesis Implications: "When Small Isn't Enough"

Support for Scale-Performance Hypothesis

This research provides strong empirical evidence for the hypothesis "When Small Isn't Enough: Why Complex Scheduling Tasks Require Large-Scale LLMs":

Clear Size-Performance Correlation

• DeepSeek R1 (7B): 0% API success → Complete failure
• Llama 3.3 (70B): 58.9% hourly success → Moderate performance
• Claude 3.7 Sonnet (~100B+): 78.5% hourly success → Good performance
• Claude Opus 4 (~1T+): 83.4% hourly success → Best reliable performance

Key Conclusions

1. Minimum Scale Threshold for Complex Optimization
2. Below 70B parameters: Unusable for production optimization tasks
3. 70B+ parameters: Usable but error-prone, requires careful validation

4. 100B+ parameters: Production-ready with acceptable accuracy rates

5. Task Complexity Drives Scale Requirements

The LED scheduling optimization task requires: - Multi-objective optimization (PPFD targets vs. electricity costs) - Complex constraint satisfaction across temporal dimensions - Precise structured output formatting (JSON) - Domain-specific reasoning about greenhouse operations

Finding: Only large-scale models (100B+ parameters) can reliably handle this combination of requirements.

1. Reliability as Critical as Accuracy

OpenAI O1's results illustrate this principle: - Accuracy when successful: Near-perfect (100% exact matches) - Practical reliability: Poor (12.5% API success rate) - Conclusion: Both scale AND architectural stability matter for production deployment

1. Practical Deployment Implications

For real-world greenhouse optimization systems: - Minimum viable scale: 100B+ parameters for acceptable reliability - Recommended scale: 1T+ parameters for optimal performance - Cost-benefit analysis: Higher API costs justified by reduced operational errors

Broader Research Implications

• Constrained optimization tasks appear to have higher scale requirements than general language tasks
• Multi-objective problems may represent a particularly challenging category requiring large-scale models
• Domain-specific fine-tuning (DeepSeek R1 example) does not compensate for insufficient base model scale
• Future research should investigate the minimum scale thresholds for different categories of optimization complexity

This research contributes to understanding when and why model scale becomes critical, specifically demonstrating that complex scheduling optimization represents a task category where scale is not just beneficial but essential for practical deployment.

Dependencies

pip install openai anthropic pandas numpy openpyxl requests scipy

Usage Examples

Generate New Test Set

from scripts.data_preparation.create_test_sets import create_test_set
test_set = create_test_set(version="v4", enhanced_instructions=True)

Run Single Model Test

from scripts.model_testing.run_model_tests import test_model
results = test_model(
    model="anthropic/claude-opus-4",
    test_set_path="data/test_sets/test_set_v3.json",
    api_key="your-api-key"
)

Analyze Performance

from scripts.analysis.analyze_performance import analyze_model_performance
analysis = analyze_model_performance("results/model_outputs/claude-opus-4_v3.json")

File Descriptions

Data Files

• test_set_v0_original.json: Original prompt version (used for DeepSeek R1 7B, caused API failures)
• test_set_v1.json: Enhanced task description with greenhouse context
• test_set_v2.json: Enhanced prompts with detailed instructions
• test_set_v3.json: Refined prompts for pure JSON output
• ground_truth_complete.xlsx: Reference optimal solutions

Scripts

• create_test_sets.py: Generates test datasets with different prompt versions
• run_model_tests.py: Executes LLM evaluation via OpenRouter API
• analyze_performance.py: Comprehensive performance analysis and reporting

Results

• model_outputs/: Raw JSON responses from each model
• analysis_reports/: Summary statistics and performance metrics
• comparisons/: Excel files comparing model vs ground truth allocations

Contributing

When adding new models or prompt versions: 1. Follow the established naming convention: {provider}_{model-name}_results_{prompt-version}.json 2. Update the analysis scripts to handle new model types 3. Document any new evaluation metrics in this README

License

This research code is provided for academic and research purposes.